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SUMMARY 


The  ASCII  CODAP  programs  developed  to  enhance  occupational  analysis  capabilities  were 
operationally  tested  on  a  number  of  example  data  sets  representing  several  recently  completed 
occupational  analysis  projects.  Using  such  examples,  the  operational  testing  compared  several 
aigorithmically  derived  solutions  with  those  actually  made  by  experienced  analysts  Feedback 
from  such  tests  were  used  to  further  refine  and  adjust  the  algorithms  used  to  identify  potential 
job  and  task  clusters.  New  displays  and  adapted  CODAP  products  needed  for  an  analyst  to 
make  final  |Ob  type  or  task  module  decisions  were  developed  and  their  utility  tested  in  actual 
use  The  overall  result  of  the  testing  suggests  that  the  advanced  technology  can  be  highly 
useful  in  assisting  analysts  to  make  realistic  decisions,  but  that  considerable  skill  and  judgment 
are  still  needed  to  properly  assess  the  significance  of  alternative  s-ts  of  jobs  or  task  modules 
With  these  new  tools,  however,  an  experienced  analyst  should  be  able  to  considerably  reduce 
the  amount  of  effort  required  to  accomplish  an  occupational  analysis  project. 
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OPERATIONAL  TESTING  OF  ASCII  CODAP  JOB  AND 
TASK  CLUSTERING  METHODOLOGIES 


I.  INTRODUCTION 

The  ASCII  CCDAP  refinements  developed  to  enhance  occupational  analysis  capability  have 
been  operationally  tested  on  a  number  of  example  data  sets  representing  several  recently 
completed  occupational  analysis  projects.  Using  such  examples,  the  operational  testing  compared 
several  algorithmic  solutions  with  those  actually  made  by  experienced  analysts.  Feedback  from 
such  tests  was  used  to  further  refine  and  adjust  the  algorithms  used  to  identify  potential  job 
and  task  clusters.  New  displays  and  adapted  CODAP  products  needed  for  an  analysis  to 
make  final  job  type  or  task  module  decisions  were  developed  and  their  utility  tested  in  actual 
use 


II.  REFINEMENT  OF  JOB  AND  TASK  GROUPINGS 

In  a  typical  occupa.,onal  study,  groups  ;dentified  as  meaningful  jobs  that  are  interpreted 
from  a  diagram  of  an  occupation  will  not  include  all  the  cases  (individuals)  in  the  sample, 
except  at  a  very  low  stage,  where  the  overlap  values  are  low.  In  previous  reports  (Mitchell 
&  Phalen.  1985;  Phalen,  Mitchell,  &  Staley,  1987),  we  reported  the  use  of  an  iterative 
nonhierarchical  cleanup  procedure  to  solve  this  problem  and  refine  the  groups.  A  sample  of 
the  resuits  can  be  seen  in  Table  1,  where  groups  from  the  hierarchical  clustering  account  for 
80%  of  the  cases  and  the  refined  groups  output  by  the  nonhierarchical  refinement  process 
encompasses  99.2%  of  the  cases  (at  iteration  6).  By  computing  the  percent  time  overlap  of 
each  case  s  job  description  with  the  mean  description  for  every  selected  group,  this  procedure 
permits  each  unclassified  case  to  be  included  in  the  group  it  most  resembles,  at  a  specified 
minimum  level  of  overlap.  The  group  vector  is  then  recomputed  and,  in  the  second  iteration 
and  beyond,  all  cases  are  compared  to  the  group  means  Cases  can  migrate  to  the  emerging 
groups  they  are  most  like,  rather  than  being  forced  to  remain  with  the  first  linkage,  as  occurs 
in  hierarchical  clustering. 


Table  i  OVLGRP-Totals  by  Iteration  (RES  811XX;  data  from  Alton,  1984) 


Iteration 

No.  classified 

No.  unclassified 

Percent 

Diagram 

2643 

660 

80.0 

1 

3298 

5 

98.8 

2 

3292 

11 

98.6 

3 

3285 

18 

99  4 

4 

3281 

22 

99.3 

5 

3279 

24 

99.3 

6 

3278 

25 

99.2 

By  looking  at  the  movement  of  cases  in  and  out  of  groups  at  each  iteration  of  the 
nonhierarchical  refinement  process,  we  can  better  understand  the  dynamics  of  the  process. 
For  example,  Table  2  shows  three  Law  Enforcement  (LE)  groups  to  illustrate  the  type  of 
changes  that  occur.  In  this  case,  we  can  track  where  the  people  go  since  the  22  members 
lost  from  the  combined  Desk  Sergeant/Patrol  group  are  the  same  22  showing  up  in  the  Desk 
Sergeant  group.  Note  that  once  these  cases  are  added,  the  Desk  Sergeant  group  tends  to 
stabilize  in  terms  of  size  and  the  variance  in  overlap  of  individuals  with  the  group  mean  drops 
considerably  (SD  =9. 2-- >7.3),  which  is  evidence  of  a  more  homogeneous  group.  The  LE 
patrolmen  group  increases  in  size  and  decreases  in  variance,  while  the  third  group  does  the 
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opposite.  Instead  of  three  groups,  we  end  up  with  two  meaningful  groups  whose  job  descriptions 
should  be  more  realistic  pictures  of  their  jobs 

Table  2.  OVLGRP-Details  of  Example  Law  Enforcement  Group 


Group 

Variable 

DGRM  GRP 

1 

2 

Iteration 

3  4 

5 

6 

GRP  594 

LE 

DESK  SERGEANT 

Grp  Size 

15 

37 

34 

33 

32 

32 

33 

No.  Lost 

- 

- 

3 

2 

1 

0 

0 

Gained 

- 

22 

0 

1 

0 

0 

1 

Mean  Ovrlp 

- 

44.6 

42.5 

42.7 

42.4 

42  5 

42.6 

SD 

- 

9  2 

7.4 

7.6 

7.4 

7.3 

7.3 

GRP  921 

LE 

DSK  SGT/PATROL 

Grp  Size 

90 

68 

38 

20 

14 

10 

5 

No.  Lost 

- 

22 

30 

18 

7 

4 

5 

Gained 

- 

0 

0 

0 

1 

0 

0 

Mean  Ovrlp 

- 

57.4 

56.2 

56.9 

55. C 

54.4 

49.2 

SD 

- 

7.2 

7.4 

8.2 

9.9 

10.5 

10.9 

GRP  785 

LE 

PATROLMEN 

Grp  Size 

12 

38 

55 

90 

84 

67 

51 

No  Lost 

- 

- 

6 

12 

29 

25 

20 

Gained 

_ 

26 

23 

47 

23 

8 

4 

Mean  Ovrlp 

- 

51.0 

52.1 

55.5 

57.2 

57.6 

57.9 

SD 

- 

10  9 

7.8 

7.4 

7.2 

6.9 

5.9 

The  reclassification  of  cases  in  iterative  stages  is  done  with  the  OVLGRP  (overlap  of  cases 
with  group  means)  program,  which  identifies  new  groups  of  cases  which  are  more  internally 
consistent  and  may  be  somewhat  easier  to  interpret  in  terms  of  core  or  characteristic  and 
distinguishing  tasks.  The  interpretation  of  the  groups  at  each  iteration  is  not  an  easy  task, 
but  better  reports  are  now  available  to  track  how  cases  move  among  job  groups  and  to  analyze 
new  job  groups. 

A  GRPMAT  (a  table  which  shows  the  migration  of  cases  from  one  job  group  to  another) 
for  the  Precision  Measuring  Equipment  Laboratory  (PMEL)  study  is  shown  as  Figure  1.  Not 
all  of  the  21  Occupational  Survey  Report  (OSR)  groups  (Aslett,  1984)  are  shown  in  this  display, 
in  order  to  simplify  the  discussion.  Note  that  the  job  types  are  shown  down  the  left-hand 
column  and  the  OVLGRP-refined  groups  across  the  top  of  this  display.  Those  cases  not 
members  of  any  group  in  OVLGRP  are  shown  as  the  first  column  (GRP  02),  and  the  marginal 
summaries  (ST  001)  report  the  size  of  each  group.  These  data  provide  some  examples  of 
what  happens  in  the  reciustering  process. 

Missing  cases  (that  is,  those  which  migrated  out  of  the  group)  differed  from  the  rest  of 
the  group  in  terms  of  their  relative  time  spent  on  duties.  They  tended  to  migrate  out  in  pairs 
which  had  some  similarity  of  duty  time  with  the  group  but  usually  performed  more  duties  as 
well. 

We  need  not  get  totally  submerged  in  the  details  of  this  process  here.  The  point  is  that 
the  reclustering  of  groufis  does  help  to  identify  distinguishing  tasks  of  various  jobs  and  can 
be  used  by  an  analyst  to  refine  his  or  her  initial  job  type  selections.  Major  groups  are 
relatively  stable  in  terms  of  their  job  descriptions,  but  some  of  the  smaller  groups  proved 
unstable  and  disappeared.  Other  small  groups  involved  in  specialized  missions,  such  as  F-15 
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equipment  maintenance,  not  oniy  proved  staple  Put  increased  in  size  in  this  iterative  regrouping 
process 


---These  are  the  Job  Types  from  the  OSR 


These  are  the  Job  Types  after  OVLGRP 


V 

Unclassified 

Grp  141 

Grp  096 

Grp  322 

Grp  238 

Grp  140 

Grp  243 

..Total 

ST  141 

95 

2 

1 

6 

112 

3T  096 

2 

160 

/ 

7 

7 

211 

ST  322 

49 

6 

1 1 

181 

ST  238 

1  7 

1 

19 

ST  140 

82 

4 

90 

ST  243 

2 

3 

5 

Total 

63 

1 Q3 

1  94 

49 

39 

.  120 

62  ... 

1513 

Figure  1 .  GRPMAT-Precision  Measuring  Equipment  Laboratory  (PMEL)  Personnel. 


The  intense  analysis  work  involved  in  interpreting  and  evaluating  the  regrouped  job  types 
can  be  facilitated  by  using  Core  Task  Analysis  and  Comparison  of  Job  Descriptions  (CORTAS) 
and  Report  of  Case  Background  Variable  Data  in  Clustering  Order  (PRTVAR)  outputs  for  the 
groups  which  need  to  be  reanalyzed  (those  which  were  not  stable  as  indicated  in  GRPMAT, 
or  where  the  analyst  wishes  an  improved  job  description).  This  approach  reduces  the  total 
amount  of  effort  needed  to  reanalyze  the  set  of  Air  Force  Specialty  (AFS)  jobs.  Another 
approach  uses  Automated  Pairwise  Comparison  of  Job  Types  (AUTOJT)  runs  to  compare  pairs 
of  groups  an  analyst  wishes  to  study;  such  AUTOJT  products  help  to  expedite  comparing  two 
groups  (input  versus  output  groups)  and  provide  a  very  quick  way  to  highlight  the  differences 
oetween  such  groups. 


III.  OPERATIONAL  TESTING  OF  THE  NEW  APPROACH 

To  test  the  new  procedures  for  nonhierarchical  refinement  of  job  and  task  clusters,  six 
AFSs  were  examined  using  this  approach.  Results  were  quite  good  and,  in  general,  replicated 
the  United  States  Air  Force  Occupational  Measurement  Center  (USAFOMC)  analysts’  judgments 
very  closely  (some  slightly  more  specific,  some  slightly  more  general).  In  Figure  1,  note  the 
distribution  of  cases  across  input  and  output  groups,  with  the  unchanged  core  members  of 
each  group  appearing  in  the  main  diagonal  Of  112  cases  in  Stage  141,  95  remain  along 
with  8  picked  up  from  other  groups;  this  represents  relatively  little  change;  thus,  there  will  be 
no  real  change  in  the  original  job  description  For  Stage  96,  160  of  the  original  211  remain, 
and  an  additional  34  are  added  from  other  groups.  A  core  of  49  people  from  the  original 
181  in  Stage  322  stayed  together  and  added  no  new  members;  the  remaining  132  cases  were 
scattered  across  8  other  groups  (not  shown) 

Stage  238  was  a  small  group  of  19;  core  tasks  which  discriminate  this  group  have  to  do 
with  calibrating  high  frequency  (HF)  counters,  and  aligning  or  troubleshooting  electronic  counters 
or  spectrum  analyzer  piug-in  units.  Seventeen  of  the  19  cases  provided  a  stable  core  around 
which  another  22  cases  clustered.  Examination  of  the  tasks  for  the  new  group  indicates  that 
the  tasks  which  discriminate  the  group  remain  the  same,  but  the  percentage  of  the  group 
performing  the  core  tasks  has  increased. 
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In  addition  to  identifying  numbers  of  individuals  who  move  or  stay  in  the  various  groups, 
an  analyst  also  needs  to  examine  the  background  data  for  these  individuals  in  order  to  fully 
understand  the  significance  of  the  various  clusterings.  Summaries  of  such  data  in  clustering 
sequence  (PRTVAR)  are  routinely  used  in  the  normal  analysis  process  Since  members  have 
migrated  among  groups,  the  normal  diagram  sequence  is  no  longer  valid;  one  would  have  to 
trace  inaividuals  through  two  sequence  numbers  case  by  case.  To  meet  this  need,  a  new 
PRTVAR  option  is  available  which  provides  a  separate  product  for  each  of  the  new  job  groups 
and  which  displays  the  data  in  original  clustering  (KPATH)  sequence. 

Table  3  reproduces  a  portion  of  a  PRTVAR  for  GRP  096  (original  Stage  096);  the  original 
clustering  sequence  for  the  cases  in  the  group  was  113  to  323.  Much  of  the  original  KPATH 
sequence  remains  intact  (in  fact,  160  of  211  cases),  with  2  cases  added  from  below  the 
original  range.  In  the  upper  KPATH  sequence,  there  is  more  mixing,  with  some  cases  migrating 
in  or  out.  It  remains  predominately  a  first  job,  with  about  half  the  members  holding  a  3-skill-level 
Air  Force  Specialty  Code  (AFSC).  There  are,  however,  some  7-skill -level  personnel  in  the 
group;  they  perform  about  the  same  number  of  tasks  and  have  a  technician  title  as  opposed 
to  calling  themselves  supervisors  or  noncommissioned  officers  in  charge  (NCOICs) 


Table  3.  PRTVAR-GRP  096;  Precision  Measuring  Equipment  Personnel 


KPATH 

No.  tasks 

AFSC 

Grade 

MAJCOM 

Base 

Job  title 

100 

95 

32470 

TSgt 

SYS 

Hanscom 

PMEL  Technician 

102 

76 

32430 

A1C 

USAFE 

Torrejon 

PMEL  Technician 

113 

73 

32450 

A1C 

SAC 

Andersen 

Prec  Meas  Equip 

114 

88 

32430 

A1C 

USAFE 

Ramstein 

PME  Spec 

115 

106 

32430 

A1C 

ATC 

Williams 

PMES 

116 

72 

32430 

A1C 

MAC 

Little  Rk 

PMEL  Technician 

117 

79 

32450 

Sgt 

SAC 

Andersen 

Precision  Meas 

118 

80 

32430 

A1C 

SAC 

Grnd  Fks 

PMEL  Spec 

H9 

6‘‘ 

Air 

AT^ 

Lowry 

PME  Spec 

319 

74 

32470 

TSgt 

SAC 

Wurtsmith 

K  1-9  Sec  Spvr 

321 

108 

32470 

SSgt 

USAFE 

Lakenhth 

PME  Tech 

324 

77 

32450 

SSgt 

MAC 

Bolling 

P‘,*r  xccy 

325 

63 

32470 

SSgt 

OAR 

HQ  AF  Cen 

PMEL  Mobile  Cal 

1093 

47 

32430 

A1C 

AAC 

Shemya 

PMEL  Tech 

The  very  positive  results  from  the  validation  ,  tsting  led  us,  in  one  of  the  monthly  CODAP 
Users’  meetings,  to  volunteer  to  use  the  automated  procedure  for  any  study  where  the  USAFOMC 
analyst  was  having  difficulty  with  an  analysis  or  where  more  than  10%  of  the  cases  were  not 
covered  by  the  identified  job  types.  USAFOMC  personnel  suggested  that  a  good  candidate 
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ioecidity  .vouid  be  the  Supply  AF$  645X  a  ;  wr>>re  junost  50G  of  the  cases  were  rot  -nr  i. 
n  ihe  analyst's  anal  set  of  selected  ,ob  :,pt*s  Caussade  19881 

initial  trials  using  the  Supply  AFS  jumped  the  coverage  of  cases  to  96  9%  which  seems 
to  be  a  dramatic  improvement  (see  Table  4i  However,  dose  examination  of  the  GRPMAT 
3 R T V A R  and  OVLGRP  oroducts  showed  an  increase  in  standard  deviations  and  a  decrease 
n  mean  /ethin-group  overlap  across  iterations  exactly  opposite  of  expectation 


Table  4  OVLGRP--Total  in  Each  Iteration;  July  1988  (AFS  645XX) 


Iteration 

No.  classified 

No.  unclass. tied 

Percent 

Diagram 

1958 

1793 

52  2 

1 

3604 

145 

96  1 

2 

3624 

127 

96  5 

3 

3627 

124 

96  7 

4 

3631 

120 

96  8 

5 

3634 

117 

96  9 

6 

3632 

119 

96  8 

Further  analysis  of  the  Supply  (AFS  645XXi  products  suggested  that  although  the  unclustered 
cases  were  now  included  in  identified  groups  the  groups  were  mostly  at  an  unacceptably  low 
level  of  internal  overlap  to  be  considered  -  a i id  |ob  types  Clearly,  the  program  was  not 
operating  appropriately  for  us  to  achieve  the  end  for  which  it  was  designed  This  unexpected 
outcome  led  to  a  complete  reexamination  of  the  software 

The  problem  appeaiS  to  have  been  with  routines  which  set  the  minimum  acceptable  overlap 
before  a  case  could  be  considered  a  member  of  a  group,  the  minimum  cutoff  was  not  operating 
as  desired  Thus,  all  cases  were  being  merged  with  some  group,  the  one  whicn  was  the 
best"  fit  for  the  case  when  compared  with  all  other  groups.  This  routine  was  rewritten  to 
effect  dual  minimum  overlap  cutoff  criteria,  one  that  is  absolute  and  one  that  varies  from  group 
to  group.  If  a  case  does  not  meet  the  dual  criteria,  it  is  rejected  for  membership  in  any 
group  at  i. iat  iteration  (and  will  be  held  for  reconsideration  at  the  next  iteration!. 

Table  5  displays  the  results  of  the  revised  program.  The  coverage  of  cases  improved  from 
about  52%  to  about  71%  of  the  cases,  without  degrading  the  job  descriptions  of  the  groups 
Tho  romaining  29%  of  the  cases  were  still  divergent  th**t  they  were  not  included. 


Table  5.  OVIGRP-Total  in  Each  Iteration  with  Improved  Program; 
October  1988  (AFS  645XX) 


Iteration 

No.  classified 

No.  unclassified 

Percent 

Diagram 

1958 

1793 

52.2 

1 

2509 

1242 

66.8 

2 

2571 

1180 

68.5 

3 

2648 

1103 

70.6 

4 

2664 

1087 

71.0 

5 

2669 

1082 

71.2 

6 

2677 

1074 

71.4 
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The  resuitm  ,  j  oups  are  much  more  acceptable  and  have  improved  face  validity  Cne 
-esuit  was  th-  oreakup  of  one  large  group  of  supervisors  into  smaller  groups  more  directly 
•eia'ed  to  tee  various  technical  areas  within  the  specialty  The  USAFOMC  analyst  involved  m 
:ms  v'udy  indicated  that  this  made  much  more  sense  in  that  the  smaller  groups  could  be 
directly  relateo  to  the  technical  subareas 

Certainly,  the  71%  coverage  is  less  than  desired  but  with  automated  technology,  it  may 
:;e  ail  mat  s  possible  for  heterogeneous  specialties  When  viewed  in  terms  of  improvement 
Over  the  original  clustering,  it  represents  a  sigmf'cant  gain  in  group  coverage  (19%)  'n 
addition  the  final  job  groups  appeared  to  be  more  realistic  and  more  -nterpretable  to  the 

analyst 


Por  most  studies  the  improved  program  should  provide  for  95%  to  99%  coverage  of  cases 
"nere  will  be  some  studies  (such  as  the  Supply  AFS),  however,  where  the  AFS  >s  so  diverse 
or  wnere  the  jobs  are  not  well  structured)  that  even  with  the  improved  program,  coverage 
tan  not  oe  more  than  70%  to  75%  We  would  maintain  that  such  complex  or  heterogeneous 
•\FSs  are  in  need  of  close  examination  They  may  represent  areas  for  possioie  reengineering 
efforts  shredouts  or  functional  reorganization; 


IV.  CONCLUSION 

Operational  testing  of  the  nonhierarchical  refinement  of  job  types  has  demonstrated  that, 
n  most  AFSs,  very  significant  improvements  can  be  made  in  the  numbers  of  members  covered 
by  identified  job  types  and  in  the  stability  of  calculated  job  descriptions.  For  some  very  diverse 
AFSs  the  refinement  procedure  will  help,  but  will  not  completely  solve  the  problems  which 
such  diversity  presents.  This  refinement  technique  is  another  very  valuable  tool  which  should 
help  occupational  analysts  improve  and  expedite  their  work 
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